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Method and apparatus for merging images into a composit image 



BACKGROUND OF THE INVENTION 
1 ■ Field of the Invention 

The present invention relates to image processing, in particular to methods and 
apparatus for registration and merging of a plurahty of overlapping two-dimensional (2D) 
5 images of three-dimensional (3D) scenes, especially in cases where the 2D images are related 
to the 3D scenes by a projective (or camera) transformation. The invention also related to 
apparatus for performing the disclosed image processing. 

2. Description of the Related Art 

10 In many fields of art and technology imaging of scenes that are too extended 

to be captured in a single camera image is important. Composite images of such scenes must 
.be merged from individual overlapping images of more limited fields of view. One example 
of where image composition is usefiil is the formation of an image of an extended scene from 
the limited fields of view of a digital camera suitable for a PC. Another example is the 

15 formation of an image of an extended region of a patient from individual x-ray images which 
are usually of more limited fields cf view. 

In many cases, such as in the previous two examples, since individual 2D 
images are related to the 3D scene by projective transformations, pairs of the individual 
images to be merged are also related to each other by projective transformations. 

20 Consequently, as part of the image merging process, it is important to identify the best 

projective transformation relating each pair of overlapping 2D images so that by use of this 
transformation the images of the pair can be brought into registration. 

Accordingly, methods have been developed and are known for determining 
such projective transformations. Typical of these methods is that disclosed in Schultz et al, 

25 1999, IEEE International Conference on Acoustics, Speech and Signal Processing , vol. 4, pp. 
3265-3268. Here, much attention is paid to automatically detemiining a plurality of pairs of 
corresponding points, one point of each pair being in each image, that is typically input in 
order to find the parameters of the projective transformation relating the images. Once the 
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pairs of corresponding points in the two image are identified, actual determination of the 
projective transformation is disclosed to be routine. 

However, merely routine determination of a projective transformation fi*om a 
plurality of pairs of corresponding points has been discovered to often not be sufficiently 
5 stable or accurate. The relevant arts need a simple and accurate method of determining such 
projective transformations. 

Citation of a reference herein, or throughout this specification, is not intended 
to be construed as an admission that such reference is prior art to the Applicant's invention of 
1 0 the invention subsequently claimed. 

SUMMARY OF THE INVENTION 

The objects of the present invention are to provide apparatus and methods 
which simply and stably determine accurate projective transformation relating two images 

1 5 from pairs of corresponding points identified in each image of the pair. 

Achieving these objects by the present invention depends on the discovery that 
projective transformation can be better determined from a plurality of pairs of corresponding 
points in both images when the numerical ranges of the coordinates of these corresponding 
points is minimized. With minimum numerical ranges of the coordinates, errors arising from 

20 terms non-linear in the coordinates are reduced in comparison with other methods which are 
ignorant of this discovery. 

The present invention minimizes these coordinate ranges prior to determining 
the projective transformation relating a pair of images. In one alternative, an original 
coordinate system is chosen in each image in advance to minimize these coordinate ranges. 

25 This choice can be made, for example, by finding a coordinate origin for which the sum of 
the radius vectors to the points is a minimum. Such a coordinate origin can be found by a 
search technique. In this alternative, the projective transformation is determined directly in 
the original coordinate system. 

In another preferred altemative, an arbitrary original coordinate system is 

30 chosen in each image. Later, a translation vector is determined so that in a translated 

coordinate system the numerical coordinate ranges are minimized. Such a translation vector 
can be determined, for example, as the average of coordinates of all the corresponding points 
in each image. In this altemative, the projective transformation is first determined in the 
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translated coordinate system, and then adjusted to apply in the original, untranslated 
coordinate system. 

In either alternative, once the projective transformation relating the two 
images of a pair of images is determined, it is applied to bring the pair of images into spatial 
5 registration, or alignment. The spatially registered, or aligned, images can then be easily 
merged by superimposition, interpolation, resampling, or so forth. 

The apparatus of this invention is configured to acquire a plurality of images 
and to carry out image merging according to the above methods. First, it can include any 
image acquisition apparatus that forms two-dimensional (2D) digital images by projection of 

10 a three-dimensional (3D) scene. Such image acquisition includes optical cameras of all sorts. 
It also includes, for example, x-ray apparatus that project an image of an object to be 
examined onto an x-ray image detection device. Also, images can simply be acquired over a 
communication link firom remote image acquisition devices or image storage devices. 

Second, the actual image processing can be performed by a suitably 

15 programmed general purpose computer, such as a PC. Alternately, it can be performed by 
specialized hardware adapted to image processing 

In detail, these objects are by the following embodiments of this invention. In 
a first embodiment, the present invention includes a method for merging a pair of 
overlapping two-dimensional (2D) images, said images being projections of a single three- 

20 dimensional (3D) scene, said method comprising: selecting at least four feature points in the 
3D scene, finding the 2D coordinate of the points in both images corresponding to the 
selected feature points, the 2D coordinates being found with respect to original coordinate 
systems in the two images, translating the original coordinate systems of the two images in 
order to substantially minimize the average coordinate ranges of the 2D coordinates found, 

25 determining the parameters of a substantially optimal projective transformation relating the 
corresponding translated coordinates in the two image, determining the parameters of the 
projective transformation for appHcation in the untranslated coordinate systems of the two 
images, and merging the two images by transforming one image according to the projective 
transformation and combining the transformed image with the other image. 

30 In a second embodiment, the invention includes an apparatus for merging a 

pair of overlapping two-dimensional (2D) images, said images being projections of a single 
three-dimensional (3D) scene, said apparatus comprising: means for obtaining a pair of 2D 
images, a processor responsive to the means for obtaining images and configured to perform 
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the methods of the first embodiment, a display for viewing the pair of images merged by the 
processor 

In a third embodiment, the invention includes an x-ray apparatus for merging a 
pair of overlapping two-dimensional (2D) images, said images being projections of a single 

5 three-dimensional (3D) scene, said apparatus comprising: an x-ray source for projecting a 
beam of x-rays through an object to be examined, an x-ray detector for obtaining digital x-ray 
images which are projections of the object, a processor responsive to pairs of overlapping x- 
ray images obtained by the x-ray detector and configured to perform the methods of the first 
embodiment, a display for viewing the pair of images merged by the processor. 

0 Li a fourth embodiment, the invention includes a computer readable medium 

comprising encoded program instructions for causing a processor to perform the methods of 
first embodiment 



BRIEF DESCRIPTION OF THE DRAWING 
1 5 Other objects, features and advantages of the present invention will become 

apparent upon perusal of the following detailed description when taken in conjunction with 
the appended drawing, wherein: 

Fig. 1 illustrates a system for practicing the invention; 

Fig. 2 illustrates an optical device for obtaining images to be processed in the 

20 invention; 

Fig. 3 illustrates a diagnostic x-ray device for obtaining images to be 
processed in the invention; 

Fig, 4 illustrates image registration and merging for a pluraUty of diagnostic x- 
ray images; and 

25 Fig. 5 illustrates an embodiment of the method according to the invention, 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

In the following, exemplary embodiments of apparatus for practicing the 
present invention are first described followed by detailed descriptions of preferred 
30 embodiments of the methods of the present invention. 



Preferred apparatus of the invention 

The present invention is preferably practiced on apparatus capable of 
appropriate processing two-dimensional digital (2D) images. Fig. 1 illustrates exemplary 
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PC-type computer apparatus 29 equipped for appropriate image processing. This apparatus 
includes PC-type computer 27 having a microprocessor for computing and image processing. 
PC 27 can also optionally include special image-processing board 28, or other similar 
hardware, for assisting or entirely performing certain image processing functions. A user 
5 employs keyboard 25 and other input devices to control the image processing according to 
this invention. Original and processed images can be displayed on monitor 20 or printed on 
hardcopy output device 24, Original and processed images can also be transferred over 
network link 26. 

Digital 2D images of 3D scenes can be obtained for input to the apparatus of 

10 this invention for processing by the methods of the present invention by any means known in 
the art. One exemplary means is to simply scan standard photographs with a digital scanner. 
Fig. 1 illustrates another exemplary means, digital PC camera 21 which includes lens system 
23 for optical imaging and CCD array 22 for conversion of an optical image to digital signals 
for input to PC 28. As illustrated, an image of a 3D scene including arrow 31 is projected 

15 onto CCD array 22 through lens system 23. 

Fig. 2 illustrates use of digital camera 32, perhaps of greater capabihty than 
PC camera 21 , which is mounted on tripod 33 for rotation about axes 34 and 35 in order to 
pan across extended scene 37. Since camera 32 can form images of only a limited part of the 
3D scene at one time, for example, of objects in cone 36 which is projected onto a digital 

20 pickup in camera 32, forming an image of entire extended scene 37 requires than multiple 
individual images be merged into a composite image by image processing apparatus 
according to the present invention. Digital camera 32 can be responsive to selected bands of 
electromagnetic radiation, for example to infrared or to visible light. 

The need to merge multiple overlapping images into a single composite image 

25 arises in many other fields of art and technology, for example in medical diagnostics. Fig. 3 
illustrates x-ray imaging apparatus 14 which forms 2D projection images of 3D patient 12, or 
of other objects to be examined. This apparatus projects x-ray beam 15 from a focal point 
within x-ray source 1, through diaphragm la, then through patient 12 and finally onto x-ray 
detector 2 which outputs a digital image signal for input to an image processor configured 

30 according to the present invention. The parameters of the x-ray image projection change 
with motions of the x-ray source and x-ray detector along the various illustrated degrees of 
freedom. 

For example, patient 12 can be longitudinally displaced along direction 1 1 on 
patient table 8 by motor means 9. The x-ray source and the x-ray detector, mounted on C- 
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arm 3 that is in turn mounted by collar 4 on support 5, are capable of coordinated rotation 
about tow perpendicular horizontal axes 12' and 12". Finally, support 5 can be longitudinally 
translated along rails 7 or rotated about vertical axis 6. The present invention is also 
applicable to x-ray apparatus with other means for jointly moving the x-ray source and the x- 
5 ray detector for rotation or translation. 

To obtain a composite image of an extended region of the patient, for example, 
of the patient's legs, a plurality of individual images formed during panning of the x-ray 
apparatus about one or more of these degrees of freedom must be merged. Fig. 4 illustrates 
in a diagrammatic fashion the formation of limited images and the merging of consecutive 

10 and overlapping images into an assembled extended image, A patient's leg 16 is shown on 
patient table 8. Vertical support 5 is moved along rails 7, so that the x-ray source is moved in 
the direction of arrow 14. As the x-ray source is moved, x-ray beam 15 is intermittently 
directed at the patient's leg. Together with the x-ray source the x-ray detector is also moved 
so as to face the x-ray source when the patient is irradiated. Whenever the patient's leg is 

1 5 irradiated a limited x-ray image is formed on the entrance screen of the image intensifies 
Thus, collection 40 is formed of consecutive images 41 1 to 41 n which mutually overlap to 
.various degrees. The overlap between sub-images depends on the displacement between 
positions of the x-ray source at the irradiation for forming said sub-images. 

An apparatus configured according to the present invention accurately merges 

20 the sub-images of collection 40 into assembled image 42, which contains a shadow-image of 
the patient's entire legs. For example, images 41n-i and 41n need to be brought into spatial 
registration before merging, mis-registration being due, for example, to planned or accidental 
changes in the orientation of x-ray beam 15 when the two projection images are formed. 
Mis-registration is reflected in area 42 of overlap where, for example, point 43n-i is at a 

25 different location than point 43n, although both points represent the same feature in patient 
leg 16. Mis-registration is corrected by determining a projective transformation the relates 
images 41n-i and 41^ so that points 43n-i and 43n are at the same spatial position. Images in 
spatial registration can be merged without blurring. 

30 Preferred methods of the invention 

Having obtained a plurality of overlapping 2D images as projections of a 3D 
scene by any appropriate image acquisition device, for example, by the devices described 
above, appropriate image processing apparatus, for example, apparatus 29, programmed to 
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perform the methods of this invention merges the overlapping individual images into a 
composite 2D image of the 3D scene. 

With reference to Fig. 5, for a pair of overlapping 2D images obtained at step 
51, these methods generally perform the following steps: first at step 52, selection of feature 
5 points in the overlapping 2D region in each image that correspond to the same feature in the 
3D scene; second at steps 53, 54, and 55, determination of a substantially optimal projective 
transformation relating the two individual images from the feature point coordinates; third at 
step 56, transformation of one image of the pair by the projective transformation; and fourth 
at step 57, merging the two images into a composite image. In the present invention, a 

10 substantially optimal projective transformation is one whose errors are due to any 

uncertainties in the input coordinate data and to any numerical errors arising in the method by 
which the transform is determined. 

Where three or more image are to be merged, pairs of images can be selected 
and pairwise merged according to the methods of the invention by following any appropriate 

15 sequence assuming the necessary image overlaps are present. For example, the images can be 
merged sequentially by initially merging the first two images, then subsequently merging the 
third image with the result of merging first two images, and so forth. Alternatively, the 
images can be merged hierarchically by initially merging the first two images, then 
subsequently merging the second two images, then merging the previous two results, and so 

20 forth. Other appropriate merging sequences can be selected to match the structure of the 
original 3D scene. 

Turning now to a more detailed description of the individual steps of the 
methods, selection of feature points in the overlapping region of a pair of overlapping 2D 
images at step 52 can, for example, be done manually. Here, a user, for example, at 
25 apparatus 29, selects N easily distinguishable points in the 3D scene that appear in both 
images. Then these feature points are identified in both images, and their coordinates are 
measured in both the images. This results in N pairs of 2D coordinates, each pair being the 
corresponding coordinates of a feature point. These are represented by the pairs: 

Ui^iuij uij , v, = (vi,/ V2,iJ , i^l,...,A^ (!) 
30 Here, the u-, are the coordinates of points in one image, and the Vj are the coordinates of 
corresponding points in the other image. 

Alternatively, the feature points may be selected automatically by the image 
processing apparatus. One automatic method proceeds by first sparsely sampling points in 
the overlapping region of the image, then using matching or correlation of locally 
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surrounding blocks of points to determine candidate pairs of sampled points that should 
corresponding to the same 3D scene point, and finally retaining only those candidates pairs 
that have sufficient surrounding image structure for accurate block matching or correlation. 
See, e.g,, Schultz et al, 1999, IEEE International Conference on Acoustics, Speech and 
5 Signal Processing , vol. 4, pp. 3265-3268. 

Another automatic method proceeds by first constructing multiresolution 
decomposition of the images by self-similar discrete wavelet transforms, then selecting 
candidate image points having local maximums of the pixel- value gradient greater than a 
threshold where the pixel-value gradient is prominent at all resolutions, and finally retaining 

10 only those pairs of candidates points that have sufficiently cross-correlated locally 

surrounding blocks of points, and thereby that should correspond to the same feature in 3D 
scene. The conditions imposed on the pixel-value gradient are to insure that there is 
sufficient image structure surrounding the candidate points for accurate cross-correlation. 
The next steps determine a single substantially optimal projective 

15 transformation that relates the pairs of corresponding points in the overlapping region of two 
images. A projective transformation best models the relation between the pairs of 
overlapping images because, as detailed above, the images are obtained by devices that 
project a 3D scene onto the 2D images. A projective transformation linking pairs of 
corresponding points is represented in the following by the following matrix equation. 

20 v,.-^^ (2) 

This transformation has the following matrix parameters which are determined by their eight 
matrix elements. 

A = h H,C = G. ,J, T = G, tj (3) 

\Ci2\ ail J 

This transformation is selected to hold for all N pairs of corresponding 2D coordinates 
25 previously selected. 

The following rearrangements develop an alternate, more compact, representation of these N 
relations. First, multiplying by • u, + 1 l^^ds to the following linear relations for the matrix 
elements. 

auuij-^an U2,i ~ c\ uij vu ~~ cj ui^i Vi,, + /i - vu (4a) 

^21 U]J + a22U2,i ~ C\ U\,i V2,t ~ C2 W2,i V2,i + ^2 " V2,i (4b) 

The following matrix form is equivalent to these relations. 
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0 0 



0 0 



'Ul,iV2,i 



-U2,iVu 
■U2,iV2J 



1 0 
0 1 



an 
a\2 
a2\ 
ail 

C\ 
C2 
tl 
tl 



(5) 



Letting P represent the 8x1 matrix of transformation parameters and letting Ui 
and Vi represent the remaining matrices in Eqn. 5, allows the following compact linear 
representations that determine parameter matrix P. 



U, P 










' V, " 










p = 











N/ 



(6) 



(7) 



Referring again to Fig. 5 in the light of Eqns. 6 and 7, next at step 53 the origin 
of the coordinate system in the two images is translated, or shifted, by vectors Uo and Vo, 
respectively, so that the following coordinate transformation obtains. 

10 U/^U/~Uo , vi = Vf-Vo (8) 

It is an important discovery on which this invention in foimded that the 
stability and accuracy of the solution for the parameter matrix, P, is considerably improved if 
the coordinate translations are chosen in order to minimize, on average, the numerical range 
of all the coordinates values of the corresponding feature points in each image. Such 
15 minimization is effective because it reduces computational errors in the terms involving the 
products of feature point coordinates appearing the matrices Ui, these product terms being 
apparent in Eqns, 4 A, 4b and 5. 

Such a minimizing translation can be determined by any means that is 
apparent to one of skill in the art. The following three relations define easily computable 
20 minimizing translation vectors, Uo and Vo, 

Wo = niin(z^,.) , vf min(v,) (9a) 



_ 1 y _ 1 y 

N i N i 



(9b) 
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min(^,)+max(^j 
2 



? vo- 



niin(v,)+max(v/) 
2 



(9c) 



Choosing the one of Eqns. 9a, 9b or 9c that leads to the smallest average range of all the 
coordinates values of the corresponding feature points, or alternatively, choosing another 
translation that leads to an even smaller average coordinate range, all N coordinate pairs are 
5 shifted by the chosen translation. 



coordinate systems in the two images are chosen so that the numerical ranges of the feature 
point coordinates are initially minimized. This choice can be done by searching for optimum 
placement of the origins of the coordinate systems in the two images. This search for the 
10 coordinate origin can seek to minimize any of Eqns. 9a, 9b or 9c, or another metric such as 
the mean square distance of the feature points from the coordinate origin. If the coordinate 
systems are so chosen, the projective transformation can be directly determined. 



matrices Ui and are determined for Egns. 6 or 7 in step 54 according to the prescription of 
1 5 Egn. 5. Since there are 8 entries in the parameter matrix, P, to be determined and since two 
equations results from each coordinate pair (see Egns. 4a and 4b), at least 4 coordinate pairs 
for 4 feature points are needed to determine P from Eqns. 6 or 7. If there are more than 4 
feature points, as is preferable, Eqns. 6 or 7 are over determined. In either case, these 
equations can be solved by known methods of numerical analysis to determine the 
20 substantially optimal projective transformation in step 55. See, z.e., Press et al, 1993, 
Numerical Recipes in C: The Art of Scientific Computing , Cambridge Univ. Press. 
One alternative solution method is to use a standard least squares method. In this method, the 
parameter matrix entries are those which minimize the following squared error function. 



25 In detail, actual matrix entries can be determined by differentiating Eqn. 10, setting the 
derivative to 0, and solving the resulting linear equations. 

Another and preferred alternative solution method is to apply a Singular Value 



In an alternative embodiment, this translation can be avoided in the original 



Returning to the preferred embodiment, after the coordinate translation, the 



^> = Eliu,-P-vf 



(10) 



Decomposition (SVD) to the matrix [Ui U2 
well-known decomposition. 




30 




(11) 



Here, R and Q are orthonormal matrices and D is the following diagonal matrix. 



PHTW000002 



11 



15 



D 



D. 0 

0 0 



with J)r = diag{di dr) 
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(12) 



Using these matrices, the parameter matrix, P can be directly determined by the following 
solution. 



P = Q^ D"' R 



V, 

V2 



d;' 0 

0 0 



with j)'^ = diag{di 



(12a) 



(12b) 



Having detennined in step 55 the projective transformation parameters in the 
translated coordinate system, in step 56 they must be altered so that the projective 
transformation can be applied in the original, imtranslated coordinate system. This alteration 
can be immediately made according to the following relations, where A, T and C are the 
10 projective transformation parameters in the original coordinate system, A', and C are the 
projective parameters in the translated coordinate system, and the coordinates (u vj in the 

two coordinate systems are related by the translation vectors {uo voj • 
C 



C - 



T-A 



Vo 



l-C-uo 
A + vo-C^ 



(13a) 
(13b) 
(13c) 



1-C -Wo 

These relations insure that the following, which represents the equivalence of the original and 
translated projective transformations, is true. 



(14) 



Finally, with the projective transformation expressed in the original, 
20 untranslated, coordinate system, the final step, step 57, of the method merges a composite 
image firom the two images. In this step, first, one of the images is transformed by the 
determined projective transformation in order to bring the two images into spatial 
registration. Next, the composite image is formed in the non-overlapping regions from the 
transformed image and the other image separately, perhaps with resampling onto a new grid 
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defining the composite image. In the overlapping region, the composite image is formed 
jfrom a combination of the transformed image and of the other image, for example, by 
interpolating the values of the points of these images and resampling onto the new grid of the 
composite image. 

The methods of this invention are readily implemented on image processing 
apparatus, for example, apparatus 29, by programming the above steps in an appropriate 
programming language, for example, C or FORTRAN. InitiaUzation and termination 
activities of such programs occur in steps 50 and 58, respectively. Optionally, numerical 
algebra packages, such as LINPACK, can be used to routine perform various of the above 
steps, such as necessary matrix multipUcation, finding the SVD and so forth. One of skill in 
the art can readily and routinely perform such programming in view of the above description. 

Computer instructions for controlling a microprocessor or an image processor 
which are generated fi-om the resulting programs can be stored on computer readable media 
for loading into a memory to control the microprocessor of the image processor to carry out 
the methods of the present invention. Such computer readable media include magnetic 
media, optical media, and even transmission over network links. 



Example 

The following example demonstrates the improved stability and accuracy 
achieved with the methods of this invention. 

First, two 2D test images representing overlapping projections of a 3D test 
scene were created. These test images were created so that they are related by a projective 
transformation having the following parameters. 

'I 0^ 
.0 1 



, C = (0 of and {-50 -llf 



Next, 27 feature points and their coordinates were determined in the 
overlapping region of the test images. The coordinate values are recorded in the following 
table. 
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TEST FEATURE PONT COORDINATES 



POINT 


vt.l,.,I-., 
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.vt/ 1 


l—l—i 
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35 
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67 


42 


17 


30 
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66 


44 
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32 
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48 
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36 
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77 


34 


27 


22 
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75 


40 


25 


28 
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38 
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26 
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36 
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37 


26 


25 


27 


67 


49 


17 


26 



Next, these pairs of corresponding coordinates were used to determine a 

substantially optimum projective transformation by the above-described methods but without 

the step of translating the coordinate system to minimize on average the numerical coordinate 

ranges of the corresponding points. This resulted in a projective transformation with the 

following parameters. 

^ 0.5099 0.0001^ 
-0.1844 0.6193. 



A = 



,C -(-0.0054 0.00027 ^z/zt/T = (-23.24 5.92/ 



10 Clearly, the A and T matrices have considerable errors. This method, which is similar to the 
direct known methods, is of questionable accuracy. 

Finally, a substantially optimum projective transformation was determined as 
above and including the step of translating the coordinate system to minimize on average the 
numerical coordinate ranges of the corresponding points. This resulted in a projective 

1 5 transformation with the following parameters. 
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f 1.1047 0.0139A , XT r 

,0.0214 1.0963j'^"^^-^°^^ 0.00067 a«JT = (-56.20 -14.997 



Clearly, the A and T matrices as determined according to the present invention are of 
substantially improved accuracy. The methods of the present invention are certainly 
relatively superior to the known direct methods. 

This comparison demonstrates the improvements achieved by the present 

invention. 



All references cited herein are incorporated herein by reference in their 
entirety and for all purposes to the same extent as if each individual publication or patent or 
10 patent application was specifically and individually indicated to be incorporated by reference 
in its entirety for all purposes. 



